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ABSTRACT 



This interim report presents the framework for classifying 
competency-based initiatives developed by the NPEC (National Postsecondary 
Education Cooperative) Working Group. Section 1 presents a structure for 
classifying competency initiatives based upon student progression into, 
within, and from postsecondary education. Key applications of competency 
definitions and assessment considered by the Working Group occur at 
intersections or transition points in the schematic. Section 2 presents the 
Working Group's current thinking with regard to defining, assessing, and 
applying competencies. Section 3 describes progress made in preparing an 
annotated bibliography of competency-based initiatives. Section 4 discusses 
the design of case studies that are being carried out to analyze current 
practices in assessing and using competencies in postsecondary education. 
Section 5 presents data and process issues (with sub-issues) identified by 
the Working Group in the form of criteria or a check list to assist 
institutions planning or implementing a competency-based initiative. Data 
issues are the following: (1) the level at which competencies are defined; 

(2) linking issues; (3) comprehensiveness issues; (4) measurement issues; (5) 
data maintenance and reporting issues; and (6) portability issues. These are 
process issues: (1) development processes; (2) involvement processes; (3) 

training; (4) implementation; and (5) costs. The case study protocol is 
appended. (YLB) 
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Data Ramifications of Competency-Based Initiatives 

Interim Report of the Working Group 
Background 



In 1997 the NPEC Working Group examining data needs related to workforce development 
policies noted an increased emphasis being placed by many on the national scene on skill 
standards, competency-based credentials, and assessment of work-based learning 
experiences, and recommended the establishment of a project focused upon the data 
ramifications of competency-based initiatives. At its meeting in January 1998 the NPEC 
Steering Committee approved a new Working Group and asked it to analyze current 
practices in defining and assessing learner competencies and to determine their utility in a 
variety of policy contexts. 

At its first meeting on September 17 and 18, 1998 members of the Working Group 
presented their experiences with a wide range of different competency-based projects and 
activities including: competency-based admissions processes; linking skills with job 
requirements; developing career transcripts; assessing critical thinking and writing skills; 
defining workforce development performance measures; and carrying out assessments at 
different points in the learning process. The Working Group discussed purposes, principles, 
pitfalls, and processes for defining and assessing competencies, and concluded that future 
directions should involve an examination of the state-of-current-practices and more in-depth 
analyses based upon case studies. 

This Interim Report presents the framework for classifying competency-based initiatives that 
has been developed by the Working Group as well as key concepts and definitions that 
evolved as applications of competencies in a wide range of policy contexts were considered. 
This report also describes progress that has been made in preparing an annotated 
bibliography of competency-based initiatives and the design of case studies that are being 
carried out to analyze current practices in assessing and utilizing competencies in 
postsecondary education. 

Defining and Classifying Competency- Based Initiatives 

At each of its first three meetings the Working Group has discussed some fundamental, 
questions: What is competence? Why is it important to assess competence? How are 
assessments used in making decisions and developing policy? The concepts resulting from 
these deliberations are presented in the next section of this report. The Working Group 
relied upon the wide range of experiences reflected in it membership to develop a “first cut" 
definition of the scope of competency-based initiatives and used that definition to guide the 
compilation of an annotated bibliography of competency-based initiatives. The Working 
Group also pursed its charge to “analyze current practices in defining and assessing learner 
competencies and to determine their utility in a variety of policy contexts” by conducting a 
series of case studies. 
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Utilizing its collective experiences and a first draft of the bibliography prepared by its 
consultant, the Working Group developed a structure for classifying competency initiatives 
based upon student progression into, within, and from postsecondary education. Key 
applications of competency definitions and assessment that have been considered by the 
Working Group occur at intersections or transition points in the following schematic. 



Figure 1 

Intersections for Consideration of Competencies in Postsecondary Education 




The elements of this classification structure where competencies become important in a 
policy or decision-making context are: upon entry into postsecondary education, at points of 
transition within postsecondary education, upon exit from postsecondary education and in 
assessing the overall effectiveness of postsecondary education. These elements define the 
scope of the competency-based initiatives project. The competency initiatives considered 
by the Working Group within each of these areas are: 

Entry Into Postsecondary Education 

(1) Competency-Based Admissions 

(2) Competency-Based Placement in Classes 

Within Postsecondary Education 

(3) Competency-Based Curricula in Specific Disciplines 

(4) General Education Competencies 

(5) Competencies at Transfer Within a Single PSE Provider (e.g., change of majors) 
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(6) Competencies at Transfer Across Multiple PSE Providers (e.g., from one institution to 
another) 

Exit from Postsecondary Education 

(7) End of Program Competencies 

(8) Postsecondary to Employment (Work Place Competencies) 

(9) Postsecondary to Graduate/Professional School 

Overall Institutional Effectiveness 

(10) Internal (Program Improvement) 

(11) External (Accountability, Performance Budgeting, Accreditation) 

The Working Group plans to continue to refine this framework, and to use it as the basic 
classification structure for the annotated bibliography and for selecting and carrying out case 
studies of competency-based initiatives. At this point the Working Group views the structure 
as the basis for defining links across the activities carried out within the project. This 
structure also parallels the work of the NPEC Student Transitions project. 



Key Concepts and Definitions 

Since its first meeting the Working Group has struggled with the question of “what is 
competence?” And, the related questions such as ”ln what ways are competencies different 
from, or the same as, learning outcomes?” “What is the relationship between the functions 
of defining competency, assessing competency and judging competency according to some 
standard?” That struggle with definitions, terminology and concepts continues. This 
section of the interim report presents the current thinking of the Working Group with regard 
to defining, assessing and applying competencies. 

Defining Competencies 

Educators use many terms interchangeably to describe learners and the results of the 
learning process including: outcomes, skills, traits, characteristics, competencies, and 
domains. Although the boundaries and meanings among these words frequently are 
blurred, it is important to strive for a clear definition of what is meant by competence. 

The Working Group suggests that competencies sit atop the apex of a hierarchy of 
experiences that have currency for the learner. Figure 2 depicts the language structure the 
Working Group has developed to differentiate among terms that appear to be used 
synonymously elsewhere in the literature. 

Traits and Characteristics: are the foundation for learning, the innate make-up of individuals 
on which further experiences can be built. Differences in traits and characteristics help 
explain why people pursue different learning experiences and acquire different levels and 
kinds of knowledge and skills. 

Skills, Abilities and Knowledge: are developed through learning experiences, broadly 
defined to include work, participation in community affairs, etc. (although the Competency- 
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Based Initiatives project is focusing on formally organized postsecondary education learning 
processes). 

Competencies: Are the result of integrative processes in which skills, abilities, and 
knowledge interact to form bundles which have currency in relation to the task for which they 
are assembled. 



Figure 2 




Characteristics of Competencies 

Skills, abilities, knowledge and competencies are often used interchangeably. For example, 
we speak of competent machinists and highly skilled machinists; competent mathematicians 
and knowledgeable mathematicians. Figure 2 presents a hierarchy to differentiate these 
terms: skills and knowledge are acquired through learning experiences; different 
combinations of skills and knowledge that one has acquired define the competencies that an 
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individual possesses; different combinations of competencies that are possessed by an 
individual are combined in carrying out different tasks. 

The term “outcomes of postsecondary education” encompasses the skills and knowledge 
acquired in the learning process. However, the outcomes of postsecondary education are 
broader than knowledge and skills and include psychosocial development; attitudes, values 
and beliefs; and civic development (See Table 2: A Taxonomy of Student Outcomes in 
Terenzini 1 ). The term “outcomes” also encompasses proxies for skills, knowledge and 
abilities including the completion of a degree or course, grades and GPA, certifications, 
employment and salary, etc. The full scope of outcomes also include those that accrue to 
postsecondary participants over long time frames, such as better health and the economic 
benefits passed on to their children. 

Competencies are a combination of skills, abilities and knowledge. Fro example, knowledge 
of the relationship between voltage, resistance and amperage (i.e., knowledge of Ohm’s 
Law, whether put in those terms or not) and skills in soldering and splicing wires, are among 
those needed to be a competent electrician. 

A given competency can be used in many different ways. For example, measuring 
distances is important to both professional golfers and surveyors. Of course, different 
distance measuring skills may be involved in carrying out these two tasks. 

Competencies have utility in the context of a task. That is, a person may have great hand- 
eye coordination, knowledge of the rules of baseball, and skill in mixing just the right 
combination of gin, vermouth and bitters, but there is no task that uses this particular 
combination of skills, abilities and knowledge. However, substitute batting skills for making 
martinis and we may have the competence needed to play the position of shortstop. 

Competence in different contexts requires different bundles of skills and knowledge. 
Leadership in a surgery suite is different from leadership on the basketball court. For 
example, motivating teammates is more important to leadership in basketball, while superior 
knowledge of the procedure is more important to leadership in surgery; but, an ability to 
effectively coordinate the roles, timing and contributions of co-workers is important to 
leadership in both contexts. The bundles of different skills and knowledge which are given 
the same label in different contexts is one reason that there is often difficulty in achieving a 
common understanding of what a given competency (like leadership) is, and then what it 
means to assess it. 

Knowing how to package the right set of competencies to effectively carry out a given task is 
in itself a competency. We sometimes refer to individuals as having great skills but are 
unable to apply them. With experience and experiment people combine gestures, phrases, 
eye contact, pace of speech, etc. in ways that allow them to give better speeches. 



1 Terenzini, Patrick T. “Student Outcomes Information for Policy Making,” NPEC, 
September, 1997. 
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Using Competencies 2 

Competency-based initiatives seek to insure that students attain specific skills, knowledge 
and abilities considered important with respect to whatever they are studying or the 
transitions for which they are preparing. 

Utilizing competencies requires the development of three distinct, but interactive 
components: a description of the competency; a means of measuring or assessing the 
competency; and a standard by which someone is judged to be competent. The following 
are examples illustrating these components in selected areas within the classification 
structure presented above: 

Entry Into Postsecondary Education (Competency-Based Admissions) 

• Defining competencies: often done by “curriculum panels” of faculty and teachers 

• Assessing competencies: various methods are used including tests; teacher ratings 
using scoring guides; and evaluation of student work samples 

• Standards for judging competence: often set by a “master panel” of secondary and 
postsecondary faculty 

Exit from Postsecondary Education-Postsecondary to Employment (Certification and 
Licensure) 

• Defining competencies: Usually done by members of the profession with a “public 
protection” perspective 

• Assessing competencies: various methods used including licensing exams, 
completion of accredited programs, successful experience in specified levels of 
practice 

• Standards forjudging competence: established in practice laws or certifications that 
have credibility with employers 

Competency-based initiatives can be important in communicating to students which 
competencies are important for them to attain; and the extent to which their learning 
experiences/efforts are meeting these expectations (i.e., the student as a consumer of 
information about competencies). 

Competency-based initiatives also can be important in communicating what people know 
and are able to do to employers or the general public (i.e., audiences external to the student 
as a consumer of information about competencies). 

Thus, a primary advantage of using competency definitions, assessments and standards is 
the process of identifying which competencies are important for students to attain, and the 
quality assurance that students actually attain them. In this context competencies provide a 
common language or “coin of the realm.” 

Difficulties are encountered as different groups use the same term for different 
competencies; or conversely, different standards for assessing the same competencies. 



2 This section draws heavily upon a communication to the Working Group from Joy McClarty 
in March 1 999. 
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Both undermine efforts to develop a common language and subsequently the utility of 
competency-based initiatives as a means of communicating with students, employers and 
the general public. 

Assessments can be carried out at any of the levels of the hierarchy shown in Figure 2. That 
is, the acquisition of skills and knowledge can be assessed; and similarly, the possession of 
competencies, and the performance of tasks involving the application of competencies, can 
be assessed. 

The case studies being carried out by the Working Group are examining how competency is 
being defined, measured or assessed, and applied utilizing standards forjudging 
competence in the wide range of areas reflected in the classification framework developed 
by the Working Group. 



The Annotated Bibliography 

Project consultants have prepared an annotated bibliography that covers the full scope of 
competency-based initiatives that are encompassed by the classification framework 
developed by the Working Group. However, the bibliography is not “exhaustive, rather it is 
"illustrative.” The bibliography focuses on the points of transition reflected in the 
classification framework and on how competencies are defined and measured. 

The consultants have also prepared a lead-in to the bibliography stating its purpose, what it 
is (representative of the full scope of competency applications) and isn’t (comprehensive), 
and how entries were selected. Similarly, a brief introduction to each section (sub-section) 
of the bibliography summarizes what is encompassed, relevant history, and extent of 
applications. 



Case Studies 



Case Study Protocols 



A series of case studies is being carried out in the project to examine how competency is 
being defined, measured or assessed, and applied utilizing standards forjudging 
competence. The case studies are being carried out in two phases so the initial studies can 
be reviewed with the NPEC Council and by the Working Group before committing to the full 
scope of studies that would encompass most if not all of the areas reflected in the 
classification framework. 

The Working Group emphasized the importance of focusing the case studies on data and 
“data ramifications.” That is, the process of establishing and assessing competencies; how 
standards are set; how agreements concerning competencies were reached; measurement 
approaches that are used; leading ultimately to how data are analyzed, aggregated, 
compared, stored and retrieved. In this context, a clearer vision of the impact that 
competencies have on decisions and policy making can be developed. It was recognized 
that it would be necessary to probe the context for competency initiatives including the 
policy purpose served, the driving forces (e.g., legislative mandates, accountability 
demands, eligibility requirements), the perspectives and influences of key players, etc. if 
data ramifications are to be appropriately examined and interpreted. 
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The case studies will utilize a common format organized around the protocols and probes. 
Further, the case studies will extend beyond merely reporting what is being done. They will 
also explore where difficulties are encountered, what strategies are succeeding, and why 
certain choices were made while other options were rejected. 

The protocols being used in the case studies are presented in Appendix A. 

Case Study Sites 

The Working Group identified potential case study sites within each area of the framework 
for classifying competency-based initiatives. Following are the possible sites identified by 
the Working Group: 

• Entry into Postsecondary Education 

(1) Competency-Based Admissions: Oregon University System; SHEEO Studies of 
Colorado, Washington and Wisconsin; New Mexico Medical School 

(2) Competency-Based Placement: Indiana University, Purdue University Indianapolis; 
ACT Compass Site; Wisconsin Data System 

• Within Postsecondary Education 

(3) Competency-Based Curricula : Aiverno College; Kings College; New Mexico Medical 
School; Columbia University Dental School; North Carolina School of Veterinary 
Medicine; Sinclair Community College; Samford University; Air Force Training Program; 
Army MOS Program 

(4) General Education Competencies: Aiverno College; Kings College 

(5) Competencies at Transfer (Within Providers) 

(6) Competencies at Transfer (Across Providers): Colorado Electronic Community 
College; North East Missouri State University; TV A University; Motorola University 

• Exit from Postsecondary Education 

(7) End of Program Competencies: TV A University; Western Governors University; New 
Mexico Medical School; Columbia University Dental School; Sun Microsystems; Cisco 
Systems; Novell 

(8) Postsecondary to Employment (Work Place Competencies): Hagerstown Community 
College; ACT Work Keys Site (Westark College); Sun Microsystems; Cisco Systems; 

Novell 

(9) Postsecondary to Graduate/Professional School 

• Overall Institutional Effectiveness 

(10) Internal (Program Improvement) 
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(11) External (Accountability, Performance Budgeting, Accreditation): State of Missouri; 

Northwest Missouri State University 

Recognizing the limits on resources currently available for the case studies, and desiring to 
see the results of a limited set of case studies before committing to the full scope of 
possibilities, the Working Group decided to carry out the case studies in two phases. 

Phase I of the Case Studies focused on refining the prototype Oregon University System 
study (including contrasting/integrating the SHEEO studies of competency-based 
admissions in Colorado, Washington and Wisconsin); Hagerstown Community College; 
Kings College; Sinclair Community College; and Western Governors University. 

The goal was to complete as much of Phase I as possible by November 1 so that this work 
could be presented at the NPEC Council meeting. Drafts of the Phase I case studies were 
also reviewed at the December 2 and 3 meeting of the Working Group. The Working Group 
provided guidance with regard to refining the Phase I case studies and discussed the 
various data and policy issues that were addressed by one or more of the institutions 
examined in Phase I. The issues identified by the Working Group are presented in the next 
section. 

The Working Group decided that the second phase of case studies would focus on a 
corporate educational program, a medical school, a military training program, the State of 
Missouri and Northwest Missouri State University; and Indiana University-Purdue University 
at Indianapolis. The final selection of four case study sites will be made by the consultant in 
consultation with the Working Group Chair after the consultant has had an opportunity to 
explore the receptivity and suitability of potential Phase II sites. 

Defining and Assessing Competence-Data and Process Issues 

As the Working Group analyzed the results of the Phase I case studies it compiled a list of 
the data and process issues that institutions encountered in implementing competency- 
based initiatives. Since a wide variety of initiatives were studied the challenges faced were 
quite different although a number of issues were common to some, if not all, of the case 
study sites. In some instances the Working Group felt that the issues were being effectively 
addressed, in other cases they illustrated the stumbling blocks that are encountered in 
defining and measuring competence. Further, some data and process issues that have 
been identified are more or less relevant in different contexts. 

The following compilation of the data and process issues identified by the Working Group 
based on Phased findings is presented in the form of criteria or a check list as opposed to 
good or questionable practices. Based upon the analyses carried out to this point it would 
be difficult to make judgments about best practices that would be uniformly applicable to all 
competency initiatives; indeed, it is unlikely that this will be possible even after the 
completion of all of the case studies and more in-depth review by the Working Group. In 
addition specific examples of data and process issues based on the case studies have not 
been included in the compilation that follows. This could likely be done although in certain 
instances it would be difficult to present the example in a positive light. Rather, the following 
data and process issues are presented in a form that hopefully would have utility as “things 
that institutions and states should carefully consider” as they planned or implemented a 
competency-based initiative. 
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I. Data Issues 



A. Level at which competencies are defined (e.g., course, major, institution) 

• Assessment processes are appropriate for the unit of analysis (e.g., course, 
major, institution) that has been chosen. 

• The unit of analysis is relevant to the target audience(s). 

• The unit of analysis is relevant to the decision-making or policy development 
context that is being addressed. 

B. Linking Issues 

• Definitions of competence are aligned with the goals of the learning 
experience. That is, the course or major, for example, is designed to result in 
the competencies identified. 

• Assessment of competence is consistent with the goals of the learning 
experience. That is, assessment processes are designed to measure the 
desired outcomes of the course or major, for example. 

• Assessment measures and results are relevant to the target audience. 

• The data reported (i.e., assessment results) have utility for decisions (e.g., 
improvements in the learning process, informing consumers). 

• Appropriate relationships are established between different assessment 
measures (and reporting procedures); for example, between “career 
transcripts” and “student development transcripts.” 

• Insights are gained by linking competence measures to activity measures, 
such as credit and contact hours. 

• Definitions and measures of competence at one level are consistent with and 
build upon those used at other levels (i.e., across disciplines, courses, 
student levels and between general education courses and courses in the 
major). 

C. Comprehensiveness Issues 

• The extensiveness of the information that is developed and maintained 
through assessment of competence reflects the right balance between costs 
and utility. For example, costs and feasibility are carefully weighed in 
designing and retaining portfolios to document competencies. 

• Cost-effectiveness trade-offs are made in supplementing or duplicating 
competency assessments; that is, in determining whether to simultaneously 
support a competence-based and traditional admissions process. 

D. Measurement Issues 

• Scales and scoring metrics or rubrics are meaningful and provide useful 
insights with regard to the competence being assessed. 

• Consistent metrics are used in assessing different competencies." For 
example, applying different scales across several competencies could be 
misleading. 
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• Results of assessments are reliable and reproducible. That is, the same 
results are achieved when assessment procedures are repeated under the 
same circumstances. 

• External data (e.g., LSAT, GRE, performance in graduate school, employer 
evaluations) are used to validate learning and competencies. 

• Precision, reliability, validity, credibility, and cost are all considered in 
selecting locally developed versus externally/vendor developed assessment 
tests and measures. 

E. Data Maintenance and Reporting Issues 

• Data about competencies are cost-effectively maintained. For example, to 
what extent should assessment results, course grades, workplace 
experiences, etc. be incorporated in a student’s career transcript or portfolio? 
Is it cost-effective to retain assessment results as part of traditional 
transcripts or in non-transcript data bases such as student follow-up files? 

• Data about competencies are organized in ways that are meaningful to target 
audiences. For example, “mastery” is documented at a level that makes 
sense to employers. Complex, multi-dimensional arrays are avoided. 

F. Portability Issues 

• Meaningful comparisons of competencies can be made across the chosen 
units of analyses; for example, secondary schools, institutions, and states. 

• Definitions of competence can be shared across the relevant units of 
analyses such as secondary schools, and colleges and universities. 

• Assessment processes and measures can be replicated across disciplines 
and institutions. 

II. Process Issues 



A. Development Processes 

• The goals of the competency development process are well defined, 
understood, and accepted. 

• The processes of defining competencies, setting standards and determining 
how competencies are measured appropriately involve all relevant 
constituents (See B below). 

• The timelines for defining competencies and standards, developing 
assessment procedures, training faculty and staff, etc. are sufficient to ensure 
credibility and success, but at the same time sustain momentum, commitment 
and interest. 

• Long term financial support for the initiative is adequate, justified and feasible 
(See E below). 

• The developmental process is portable; that is, it can be replicated at multiple 
institutions, or across multiple disciplines, if necessary. 

• Steps are taken to ensure that the developmental process is credible to target 
audiences (e.g., students, faculty, employers, legislators, accrediting bodies). 
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B. Involvement Processes 



• Faculty are involved in the development of competence definitions, standards 
and the development of assessment procedures. Careful consideration is 
given to the formal mechanisms for gaining faculty input, how faculty are 
named to carry out these responsibilities, how and when the faculty as a 
whole are involved/informed, etc. 

• Incentives and support are provided to ensure effective faculty contributions. 
For example, faculty are given released time to compensate for their efforts. 
Also, faculty are given recognition and assurances of ownership. 

• Students are encouraged to use information about competencies in making 
choices and in “self-assessment.” Student awareness of their competencies 
is an important objective of the overall initiative. 

• Student perspectives are appropriately considered in the development of 
competence definitions and standards. As individuals who use information 
about competencies students have input concerning what is included and 
excluded from specifications of competency goals. 

• Involvement in the development of competence definitions, standards and the 
development of assessment procedures is recognized as a important step in 
gaining “buy in’ and support of key constituents. 

C. Training 

• Faculty and staff are provided training in measuring competencies (e.g., in 
scoring and ranking techniques). 

• Users are helped to effectively apply information concerning competencies in 
carrying out their decision-making and policy development responsibilities. 

For example, they are trained in how to interpret assessment results, how to 
consistently apply standards, and how to analyze the relationships between 
learning activities and competencies. 

D. Implementation 

• Competency assessments are reported to the target audience in a form that is 
meaningful and conducive to use in the intended decision-making or policy 
development process. 

• All participating departments or institutions are capable of carrying out the 
planned definitional and assessment processes. For example, smaller 
institutions have the resources and capabilities to carry out assessments, 
maintain records, etc. in a manner envisioned by the overall plan. 

• Checks and balances are built into the process to ensure reliability of data. 
Procedures for verification of scoring, ranking, and reporting are utilized. For 
example, efforts are made to ensure that faculty, admissions staff, etc. use the 
same definition of competence level for a given skill. 

• Efforts are made to ensure the credibility of results. For example, timely 
information about definitions, standards, and assessment methodologies is 
provided to users. 

• The initiative is periodically evaluated and the appropriate constituents are 
involved in and informed about the evaluation. 
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E. Costs 



• The resource requirements to support the competency initiative have been 
analyzed and realistically projected. These resource commitments are 
justified in terms of the expected benefits and overall priorities of the 
institution. 

• The budget needed to carry out the competency initiative is feasible and 
sources of support have been identified. 

• Seed money requirements, external sources of seed money, the costs of 
“front end” incentives, faculty development costs, the costs of operating dual 
systems, etc. have been realistically projected. 

As the Phase II case studies are completed this compilation of data and process issues will 
be refined and extended. 

Integrating the Results into a Final Report 

Upon completion of the case studies and the bibliography the Working Group will have 
examined a wide range of competency-based initiatives. The results of the case studies will 
have provided insights with regard to processes for establishing and assessing 
competencies, what strategies are successful and where difficulties are encountered. The 
case studies should provide a basis for drawing conclusions about principles and practices. 
In addition the Working Group will have considered key concepts and definitions related 
competencies during the course of its first meetings. These results will be woven together 
into a final report. 

Clearly, major components of the final report will be based on the case study reports and the 
annotated bibliography. It is likely that an introductory section(s) would draw upon the key 
concepts and constructs considered by the Working Group during the course of its 
meetings. And a concluding section(s) would focus on the findings of the project and 
address principles and practices based upon the data and process issues identified in the 
preceding section. 

The Working Group intends that its report would have an important role in a number of 
areas: 

• To show the importance of competency initiatives in broader contexts such as economic 
development, technology-based instruction, and industry interests in competency 
certifications. To address questions of why competencies are important and likely to 
become more important. 

• To reach important target audiences (e.g., presidents, board members, legislators) that 
are not likely to avail themselves of other work in the area of competency definition and 
assessment. 

• To address terminology and definitional issues: What is a competency? How does it 
differ from a student outcome, skill, or ability? What is the relationship between 
performance, assessment, and standards? 
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. To establish a set of “principles” or criteria that could be used by institutions or states 
that are planning or implementing competency-based initiatives. 

Initially, the Working Group discussed addressing these objectives through an early, very 
brief publication or “primer.” However, the Working Group decided that these objectives 
could best be achieved within its final report as outlined above. It may be possible to 
separately disseminate an executive summary of the final report as a “primer” or project 
overview. 

The target audience for the final report would be academic leadership, particularly those 
administrators and policy-makers at the state and institutional levels who are involved in 
implementing or planning competency initiatives. 

Timelines 

July 1 999 - December 1999: Phase I Case Studies and Bibliography completed 
November 1-3: Work of the project presented at the NPEC Council meeting 
December 2-3: Third Working Group meeting 

January 2000: Project status and proposals for future work presented to the NPEC Steering 
Committee 

April 2000: Drafts of Phase II Case Studies completed 
July 2000: Fourth Working Group meeting 
September 2000: Final Project Report completed 
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Appendix A 

CASE STUDY PROTOCOL 



Thank you for meeting with me today. As you know, I am interviewing you as part of a 
NPEC [explain NPEC if needed] project studying the data ramifications of competency- 
based initiatives. 

1. What is jour name and position? How have you been involved in [NAME OF COMPETENCY- 
BASED INITIATIVE]? 

2. Why did you decide to use a competency-based model? 

Probe: What did you want to change? 

3. Describe how competencies are used in the [NAME OF COMPETENCY-BASED 
INITIATIVE] process. 

Probe: Were competencies particularly useful? If yes, how? If not, why not? 

Probe: Contrast the use of competencies with how the process was traditionally done. 
Probe: Is the competency-based process optional or required? 

4. How were the competencies developed? 

Probe: Were competencies created from scratch or were they based on work done 
elsewhere? 

Probe: Are there any particular problems with “home-grown” competencies? 

Probe: How do you know if you have selected the appropriate competencies? 

Probe: At what level of specificity are the competencies expressed? Are they 

expressed numerically or qualitatively? Do they include cognitive and affective 
competencies? Are attitudes or dispositions included among the competencies? 
Probe: Can the competencies ever change? What process is involved? 

5. Who provided leadership in the development of the competencies? What groups were 
involved? Do these groups remain responsible for the competencies? What challenges 
did you encounter in this development process? What worked particularly well? 

6. How are the reliability and validity of these competencies ensured? 

Probe: What data are being gathered? Do you have examples of data that you could 
share with us? 

Probe: Who evaluates the quality of the data? Or, Who decides whether levels of 
reliability and validity are good? 

7. How are you measuring the identified competencies? 

Probe: What types of assessments are used? 

Probe: Do scores ever “expire”? Are they “date-stamped"? 

Probe: What challenges do you encounter in this measurement/assessment process? 
Probe: What works particularly well or what are your successes with assessment? 
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8. Are competencies linked structurally throughout the institution or organization? How? 
(For example, do program competencies guide course competencies?) 

9. Are these competencies transportable? How do you ensure transportability? 

10. What advice would you provide to colleagues at other colleges who want to develop and 
implement a competency-based curriculum? (What policies needed to be altered in 
order to accommodate using competencies?) 

1 1 . What would you do differently if you were to start over? 

12. Is there anything else you would like to share about this competency-based initiative? 
Thank you for your time. We appreciate your willingness to be interviewed. 
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